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PREFACE 


This  report  documents  work  completed  on  the  study  of  technology 
transition  as  a  process  for  improving  training  methods  and 
technology  from  Armstrong  Laboratory  to  user /customers. 
Specifically,  it  examines  the  application  of  a  low-cost,  part-task 
trainer  in  three  pilot  communities.  Results  of  observation  of 
device  usage,-  data  from  interviews  and  questionnaire  surveys  of 
users,  and  training  effectiveness  data  are  synthesized  and 
contained  in  the  report.  Current  aircrew  training  areas  and 
practices  which  may  benefit  from  this  research  include: 

1.  Unit-level  training  requirements  and  objectives 

2.  Application  of  simulation-based  training  devices 

3.  Training  technology  transition 

The  work  was  accomplished  under  Work  Unit  1123-25-15,  Unit  Level 
Training  Research  Applications.  The  project  scientist  was  Or. 
Bernell  J.  Edwards. 
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LESSONS  LEARNED  FROM  THE  DEVELOPMENT  AND  TRANSITION 
OF  THE  AIR  INTERCEPT  TRAINER  (AIT) 

EXECUTIVE  SUMMARY 


This  report  documents  lessons  learned  from  an  effort  to 
transition  the  air  intercept  trainer  (AIT)  as  new  training 
technology  from  the  Armstrong  Laboratory,  Aircrew  Training  Research 
Division  (AL/HRA)  to  Air  Force  users.  The  development  and 
transition  period  spanned  more  than  six  years  during  which  design 
concepts  evolved  permitting  the  fusion  of  emerging  technology  with 
changing  user  interests  and  needs.  This  report  deals  more 
specifically  with  findings  from  the  perspective  of  AIT  users, 
rather  than  from  an  engineering  development  point  of  view,  although 
some  of  the  latter  aspects  have  been  included.  The  report  presents 
a  summary  of  the  major  lessons  learned  from  the  effort  as  contained 
in  this  executive  summary.  For  readers  interested  in  additional 
detail  on  findings,  this  report  provides  background  and  elaboration 
of  findings,  beginning  on  page  9. 

The  lessons  learned  are  grouped  and  presented  in  order  of  the 
development  and  transition  of  the  AIT  to  three  groups  of  users. 
The  first  user  group  was  the  Air  National  Guard  (ANG) ,  162nd 
Training  Squadron  (TS) ,  Tucson,  Arizona.  The  second  period  of 
transition  involved  the  Air  Combat  Command  (ACC) ,  58th  Tactical 
Training  Squadron,  Luke  AFB,  AZ  for  F-16  initial  qualification 
pilot  training.  The  third  period  of  development  represented  an 
extension  of  the  AIT  training  concept  into  the  operational  F-16 
community  of  the  Air  Force  Reserve  (AFRES) .  This  third  period  also 
included  expanded  use  of  the  trainer  by  the  Air  National  Guard. 
Each  of  these  development  and  transition  periods  yielded  a  variety 
of  data  and  lessons  learned  upon  which  subsequent  improvements  in 
the  device  were  made  possible. 

Finally,  a  review  of  the  major  issues  investigated  and  the 
relevant  lessons  learned  for  each  issue  is  presented  as  a 
conclusion  to  this  report. 

Trainer  Description 

The  original  air  intercept  trainer  prototype  was  developed  as 
a  technology  testbed  device  to  support  Air  National  Guard 
conversion  training  from  the  A-7D  to  the  F-16A  aircraft.  The 
purpose  of  the  AIT  was  to  train  pilots  to  perform  the  beyond- 
visual-range  portion  of  air  intercepts  which  accounts  for  about  90% 
of  these  maneuvers.  The  AIT  consisted  of  a  quasi-cockpit  equipped 
with  a  head-up  display  (HUD) ,  a  radar  electro-optical  display 
(REO) ,  throttle  quadrant,  and  side-stick  controller.  The  trainer 
simulated  basic  aerodynamic  and  selected  avionics  capabilities  of 
the  F-16  aircraft  required  in  performing  intercepts.  AIT 
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development  evolved  through  three  basic  models  or  configurations 
spanning  approximately  six  years. 

The  original  AIT  prototype  simulated  selected  aspects  of  the  F- 
16A  model  aircraft  and  was  later  modified  to  simulate  the  F-16C 
model  aircraft.  The  original  trainer  permitted  the  student  limited 
control  of  HUD  and  radar  modes  of  operation,  but  did  not  permit 
pilot  control  of  simulated  flight.  Rather,  a  computer  model  of  an 
idealized  intercept  was  flown  for  the  pilot  by  the  trainer. 
However,  soon  after  its  introduction  in  the  squadron,  the  prototype 
was  upgraded  to  provide  realtime  flight  controls  for  the  pilot. 
Training  software  in  the  AIT  contained  practice  scenarios  of 
varying  complexity.  Using  these  scenarios  pilots  learned  to 
acquire  and  intercept  simulated  enemy  aircraft. 

A  second  major  upgrade  in  the  AIT  occurred  about  two  years 
following  its  introduction  to  the  ANG.  This  version  included 
expanded  and  improved  training  scenarios  including  radar  targets 
with  maneuvering  capabilities,  improved  feedback  and  debriefing 
capabilities  for  training  purposes,  and  a  forward  field-of-view 
display  of  the  flight  environment. 

A  third  version  of  the  AIT  followed  about  two  years  after 
version  two.  This  AIT  incorporated  enhancements  intended  to 
support  the  training  requirements  of  operational  units  in  the  Air 
Force  Reserve.  Enhancements  included  more  capable  maneuvering 
targets,  more  complex  training  scenarios,  and  the  capability  to 
electronically  link  or  "network"  two  AITs  so  that  pilots  could 
simulate  two-ship  intercepts.  For  a  more  complete  description  of 
the  AIT,  see  Boyle  and  Edwards,  1992  (Fig.  1) . 


Technology  Transition  Issues 

From  the  outset,  the  intention  of  the  laboratory  was  a  joint 
effort  with  users  not  only  to  improve  the  training  device  itself, 
but  also  to  understand  and  improve  the  capability  of  AL/HRA  to 
transition  technology  to  users.  In  general,  the  effort  included  a 
continuing  and  long-term  dialog  with  users.  In  attempting  to 
understand  the  process,  several  component  issues  were  identified 
and  incorporated  to  guide  collection  of  meaningful  data.  These 
included  user  acceptance  of  the  device,  usage  methods/patterns, 
design  features  and  options,  training  capability  and  effectiveness, 
device  reliability,  technology  development,  and  technology 
transition. 


Data  Collection 

Data  were  collected  over  an  extended  period  of  time  during 
which  the  AIT  was  installed  as  a  training  testbed  with  various 
users.  Types  of  data  collected  included  observations  made  by 
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AL/HRA  investigators  at  testbed  locations,  questionnaires, 
interviews,  and  two  experimental  studies  of  training  effectiveness. 

Lesson  Learned  from  the  ANG  Experience 

The  first  AIT  prototype  was  installed  at  the  ANG  Tucson  to 
support  the  conversion  course.  It  was  adopted  as  a  standard  part 
of  the  training  syllabus.  During  the  ensuing  two-year  period,  the 
AIT  as  applied  by  the  user  to  the  ANG  pilot  training  program  was 
the  subject  of  intensive  study  by  AL/HRA. 


1.  User  Acceptance  and  Training  Evaluation  Results.  The  training 
capabilities  and  utility  of  the  AIT  were  well  accepted  by  the  Air 
National  Guard  training  squadron  and  the  device  became  a  key 
training  resource  during  the  squadron's  conversion  to  the  F-16A 
aircraft  from  the  A-7D  aircraft. 

Laboratory  investigators  used  observational  and  interview 
techniques  to  evaluate  device  training  utility.  Extensive  study 
of  the  methods  used  by  instructors  in  training  students  with  the 
device  was  conducted.  Interviews  of  pilots,  instructors,  and 
students,  were  also  conducted.  Analysis  of  the  data  collected 
clearly  showed  a  high  rate  of  user  acceptance  of  the  trainer  within 
the  squadron.  Both  student  and  instructor  opinion  was  positive 
regarding  the  training  utility  of  the  device  for  conversion 
training.  Instructors  liked  the  unique  capability  afforded  by  the 
device  to  manipulate  the  avionics  systems  via  the  instructor 
console  in  order  to  teach  radar  employment  and  intercept  geometry. 
Students  appreciated  the  opportunity  to  become  familiar  and 
proficient  on  basic  skills  such  as  switchology  before  attempting 
intercepts  in  the  aircraft. 


2.  Familiarisation  Training.  Students  required  considerable 
instructor-assisted  training  to  learn  basic  operation  of  the  AIT. 
However,  once  students  were  familiar  with  the  equipment  and  could 
"navigate"  within  the  software,  self-paced  practice  on  the  device 
became  a  realistic  option. 

The  original  version  of  the  operator's  manual  provided  with  the 
prototype  trainer  proved  to  be  inadequate  in  scope  and  content  for 
purposes  of  enabling  students  to  operate  the  trainer.  Instructors 
found  it  necessary  to  devote  about  an  hour  per  student  to  insure 
adequate  familiarization.  Additional  instructional  materials  were 
developed  to  supplement  the  operator's  manual  and  were  incorporated 
in  training  to  standardize  student  familiarization  procedures. 


3.  Svmboloqv  Interpretation  as  an  Enabling  Skill.  ANG  pilots 
required  substantially  improved  training  in  HUD/radar 
interpretation  skills  as  a  lead-in  to  the  AIT  phase.  Lack  of  these 
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•nab ling  skills  eaussd  AIT  training  to  b«  much  less  efficient  than 
it  should  have  been  during  the  first  several  classes.  This  led  to 
revision  of  the  training  syllabus  to  provide  more  AIT  time  for 
basic  skills  development  and  eventually  led  to  the  development  of 
special  purpose  computer-based  training  materials. 

A  PC-based  test  was  developed  by  AL/HRA  which  permitted 
measurement  of  HUD/radar  symbology  interpretation  skills.  The  test 
also  provided  a  means  to  determine  when  a  student  had  acquired  a 
sufficient  level  of  skill  to  begin  AIT  training.  For  the  first 
several  classes,  results  of  testing  showed  generally  low 
interpretational  skills  both  before  and  after  AIT  training. 

As  a  result,  a  drill-and-practice  program  was  developed  and 
added  to  the  test  to  aid  pilot  acquisition  of  symbology  skills 
prior  to  AIT  training.  This  program  substantially  improved  the 
efficiency  of  AIT  training. 


4.  Lack  of  Visual  Display.  The  prototype  AIT  was  not  equipped 
with  a  forward  view,  out-of-the-cockpit,  visual  display.  While  the 
initial  reaction  of  ANG  student  pilots  to  this  limitation  was 
negative,  training  effects  were  positive.  The  resulting  "forced" 
reliance  on  HUD/radar  information  during  training  produced 
substantial  skill  gains  in  intercept  geometry  and  situational 
awareness,  as  reported  by  instructor  pilots. 

At  first,  many  pilots  reported  feeling  uncomfortable  with 
having  to  "fly"  the  device  based  only  on  information  from  the 
HUD/radar  displays.  However,  with  additional  practice,  pilots 
found  little  difficulty  controlling  simulated  flight  in  the  AIT. 
Instructors  reported  the  HUD/radar  skills  thus  acquired  developed 
rapidly  and  transferred  effectively  during  aircraft  training. 


5.  Fidelity  Limitations.  AMG  pilots  found  the  lack  of  pilot- 
controllable,  simulated  flight  of  the  trainer  unacceptable.  As  a 
result,  AL/HRA  promptly  developed  and  retrofitted  enhanced  software 
to  provide  acceptable  flight  controls.  Device  training  utility  and 
pilot  acceptance  clearly  increased  following  this  upgrade.  The 
resulting  lesson  for  AL/HRA  was  that  any  aircraft  capabilities 
included  in  the  simulation  must  be  functionally  equivalent  to  the 
aircraft.  Or,  if  present  in  the  device,  they  must  be  of  such 
obviously  low  fidelity  that  pilots  are  not  led  to  equate  trainer 
capabilities  with  those  of  the  aircraft. 


6.  Intercept  Performance  Measurement.  Assessment  of  pilot 
performance  of  intercepts  was  better  accomplished  using  well- 
developed  debriefing  feedback  techniques  rather  than  a  computer¬ 
scoring  approach.  The  development  of  a  linear  scoring  system  which 
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assigned  weighted  values  for  various  intercept  components  proved 
inadequate  for  purposes  of  training  evaluation. 

As  a  result  of  this  lesson,  subsequent  approaches  to 
performance  evaluation  during  AIT  training  have  emphasized  methods 
to  improve  the  feedback  to  the  pilot  by  providing  enhanced  graphic 
representations  of  intercepts  to  support  debriefing  activities 
following  training.  A  subsequent  evaluation  of  AIT  training 
effectiveness  employed  ratings  of  intercept  performance  by  expert 
pilots  rather  than  the  computerized  model. 


7.  Training  Effectiveness: _ Empirical  Data.  Pindings  from  an 

initial  AIT  empirical  evaluation  were  inconclusive  relative  to  AIT 
training  effectiveness. 

In  addition  to  opinion  data  collected  from  instructors  and 
students  after  the  prototype  device  was  installed  at  the  squadron, 
AL/HRA  conducted  an  experimental  study  to  assess  the  transfer 
effectiveness  of  AIT  training  to  the  F-16  simulator.  Problems  in 
experimental  design  and  methods  were  judged  to  be  responsible  for 
inconclusive  results.  Lessons  learned  from  this  study  were  applied 
to  improve  a  subsequent  training  effectiveness  evaluation  of  the 
AIT. 


8.  Prototyping  Methodology.  The  development  and  delivery  of 
timely,  yet  state-of-the-art  training  technology  directly  from 
AL/HRA  to  the  user  presented  a  dilemma.  The  prototyping  process 
had  to  accommodate  timely  infusion  of  new  technology,  otherwise, 
the  trainer  would  be  obsolete  upon  delivery.  However,  if 
technology  infusion  dominated  development,  the  effort  would  become 
consumed  "chasing"  technology  so  that  a  usable  device  would  never 
be  delivered.  A  balance  was  reached  by  developing  a  two-step 
prototyping  approach  which  made  possible  timely  infusion  of 
technology  into  the  trainer  design.  A  first-stage  device  was  used 
for  engineering  development  to  work  out  problems  with  new 
technology;  a  second-stage  device  was  used  by  pilots  to  test 
software  changes  and  to  determine  when  design  changes  were  mature 
enough  to  be  fielded. 


Lessons  Learned  from  the  ACC  Schoolhouse  Experience 

AL/HRA  researchers  installed  an  upgraded  version  of  the  AIT  at 
the  58th  Tactical  Training  Squadron,  Luke  AFB  approximately  two 
years  following  its  introduction  at  the  ANG,  Tucson.  The  AIT 
delivered  to  the  Luke  schoolhouse  incorporated  substantially  more 
training  capabilities  than  the  original  ANG  version.  Enhancements 
included  instructor-configurable  training  scenarios,  improved 
aerodynamics  and  handling  characteristics,  and  the  outside-the- 
cockpit  visual  display  previously  described.  The  trainer  was 


integrated  into  the  initial  qualification  training  course  for  F-16 
pilots.  The  B-Course  (for  pilots  coming  directly  from 
undergraduate  pilot  training)  was  modified  to  incorporate  the  AIT 
as  an  experimental  adjunct  to  the  syllabus.  AL/HRA  monitored 
usage  and  management  of  the  AIT  in  the  B-course  for  about  two  years 
and  collected  data  to  evaluate  training  device  effectiveness. 


l.  Utility  and  Training  Effectiveness.  The  AIT  was  well  accepted 
in  the  sehoolhouse  community  as  an  introductory  level,  air-to-air 
training  device.  it  filled  an  important  training  need  in  the 
course.  The  effectiveness  of  the  AIT  and  its  acceptance  in  the 
sehoolhouse  were  the  result  of  extensive  training  analysis  and 
design,  technology  development,  and  long-term  efforts  to  improve 
device  training  qualities,  including  the  application  of  lessons 
learned  from  experience  with  the  ANG. 

The  training  analysis  and  design  was  a  long-term  effort  that 
also  successfully  incorporated  the  use  of  subject-matter  experts 
(SMEs)  as  consultants  in  the  development  and  transition  process. 
The  product  evaluation  involved  extensive  observation  of  training 
activities,  assessment  of  pilot  opinion,  and  collection  of  training 
data  in  the  squadron.  A  formal  evaluation  of  the  AIT  in  the  B- 
course  demonstrated  the  training  effectiveness  of  the  trainer. 


2.  Self-Paced  Training  capability,  student  pilots  were  able  to 
successfully  use  the  training  menus,  instructional  features,  and 
practice  scenarios  without  the  assistance  of  an  instructor  pilot. 
Students  generally  preferred  the  self-paced  mode  of  training  on  the 
AIT  as  a  means  toward  developing  intercept  shills.  Instructors 
also  accepted  the  self-paced  mode  of  use. 


3.  Instructor  Acceptance,  instructor  pilots  accepted  the  AIT  as 
a  useful  training  resource.  This  was  attributable  to  the  A$T's 
training  software  capabilities,  compatibility  with  syllabus 
objectives  and  content,  and  the  convenience  to  students  afforded  by 
the  presence  of  the  AIT  in  the  academic  squadron. 


4.  Adoption  of  the  Trainer.  Data  from  the  sehoolhouse 
evaluation,  both  from  user  opinion  and  empirical  training  data, 
pointed  clearly  to  the  effectiveness  of  the  AIT  and  the  needror 
adoption  into  the  syllabus.  However,  the  decision  to  implement  the 
trainer  formally  was  delayed  with  the  result  that  the  second 
sehoolhouse  (MacDill  APB,  FL)  did  not  receive  training  benefits 
from  the  AIT  for  some  two  years  following  completion  of  the 
empirical  study  at  Luke.  Stronger  advocacy  of  the  AIT  by  AL/HJ2A 
might  have  accelerated  acquisition  of  the  second  sehoolhouse 
device. 
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Lessons  Learned  fron  the  AFRES  Experience 

As  a  logical  outcome  of  the  considerable  success  of  the  AIT  in 
the  ANG  and  ACC  schoolhouses,  an  attempt  was  made  to  use  AITs  in 
the  operational  F-16  squadrons  of  the  Air  Force  Reserve.  Trainers 
were  upgraded  to  include  more  sophisticated  targeting  capabilities 
and  training  scenarios.  Two  AITs  were  networked  together  so  that 
pilots  could  simulate  two-ship  operations.  AITs  were  installed  at 
several  AFRES  squadrons  over  a  period  of  about  two  years  for 
evaluation  of  training  utility.  The  following  items  summarize  the 
lessons  learned  from  use  of  the  trainers  in  the  AFRES  squadrons. 


1.  Trainer  Utility  for  Operational  Pilots.  For  experienced  F-16 
pilots  in  the  AFRES  operational  squadrons ,  the  training  utility  of 
the  AIT  was  limited.  Even  with  enhancements  in  the  capabilities  of 
the  trainer,  pilots  in  these  squadrons  were  able  to  realize 
relatively  minor  training  benefits  from  the  AIT  such  as  basic 
switchology  skill  refreshment. 


2 .  Basic  Skill  Refraahmont  Value.  The  training  utility  of  the  AIT 
for  switchology  skill  maintenance  corresponded  with  pilot 
experience. 

Less  experienced  operational  pilots  derived  some  benefit  from 
the  AIT  for  switchology  skill  refresh.  Therefore,  units  with 
relatively  large  numbers  of  less  experienced  pilots  benefitted 
somewhat  more  from  the  AITs  than  did  squadrons  with  proportionately 
higher  numbers  of  pilots  with  more  experience. 


3.  Fidelity.  Limitations  in  AIT  fidelity  and  training 
capabilities  which  were  acceptable  in  the  schoolhouse  were  found  to 
be  unacceptable  to  operational  pilots  in  AFRES  units,  particularly 
in  units  where  pilot  experience  was  high.  Data  collected  in  these 
squadrons  showed  that  once  a  pilot  had  accrued  about  300  hours  in 
the  jet,  the  value  of  AIT  training  became  negligible. 

Fidelity  was  a  factor  in  keeping  the  cost  of  the  AIT  to 
acceptable  level.  Shortcomings  in  some  aspects  of  AIT  flight 
control  fidelity  were  of  minor  consequence  in  learning  basic  skills 
and  intercept  concepts.  However,  these  proved  to  be  distracting  to 
experienced  pilots.  Design  of  intercept  scenarios  and  the  limited 
capabilities  of  computerized  targets  proved  to  be  unchallenging  to 
AFRES  pilots  and  therefore  basically  unacceptable  for  training 
purposes . 


4.  Design  Enhancements.  To  support  AFRES  F-16  operational  unit 
training,  required  device  capabilities  extended  well  beyond  those 
of  the  AIT.  In  'view  of  the  results  obtained  using  the  AIT  in 
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AFRES,  the  magnitude  required  changes,  and  their  associated  costs, 
the  reasonable  choice  would  be  to  design  a  new  device,  rather  than 
attempting  extensive  modification  of  the  AIT. 

The  two  major  areas  of  device  enhancement  would  be  in 
simulation  fidelity  and  training  capability.  Device  fidelity 
development  would  include  aircraft  aerodynamics,  engines,  cockpit 
environment  (for  relevant  controls  and  displays) ,  radar,  HUD,  and 
relevant  weapons.  Major  areas  of  training  capability  development 
include  target  maneuvering  capabilities  to  levels  which  challenge 
the  knowledge  and  skill  of  experienced  pilots. 


5.  Use  of  Two-8hip  Mode.  Although  the  networking  capability  of 
two  AITs  for  two-ship  operations  was  developed  as  a  training 
enhancement  for  the  AFRES  squadrons,  pilots  in  these  units  made 
little  use  of  the  two-ship  feature.  Related  to  this  finding  was 
the  fact  that  pilots  preferred  to  use  the  AIT  in  a  "solo"  mode. 
Pilots  also  reported  having  technical  problems  using  the  two-ship 
mode. 

It  had  been  originally  hoped  that  the  addition  of  the 
"networked"  AIT  cockpits  would  compensate  for  other  training 
limitations  and  that  pilots  would  be  motivated  to  use  it 
extensively.  However,  this  turned  out  not  to  be  the  case. 


6.  Trainer  Reliability  and  Technical  Service.  Better  technical, 
service  to  users,  particularly  during  the  initial  period  of  use  at 
the  unit,  could  have  alleviated  trainer  reliability  problems  and 
promoted  better  user  acceptance  of  AITs  in  the  squadrons.  This  was 
especially  true  for  the  first  AFRES  squadrons  who  received  the 
AITs.  The  operational  reliability  of  original  AIT  had  been 
exemplary.  However,  as  the  trainer  evolved  and  its  complexity 
increased,  so  did  reliability  problems.  By  the  time  the  AITs  were 
fielded  to  AFRES  units,  reliability  problems  had  become 
considerable  and  required  special  attention. 

Analysis  of  AIT  repair/maintenance  procedures  revealed  that  two 
relatively  simple  changes  would  improve  service  to  the  field:  (a) 
provide  a  single  point  of  contact  for  maintenance  for  each  field 
user  and  (b)  provide  more  frequent  contact  with  field  users. 


7.  Technology  Transition.  Fully  transitioning  the  prototype  AIT 
from  the  laboratory  to  the  user/manufacturer  was  not  possible 
because  of  inability  to  assume  responsibilities  for  software 
development,  upgrades,  and  maintenance.  As  a  result  technology 
transition  was  incomplete;  AL/HRA  had  to  retain  responsibility  for 
software  development  and  maintenance. 
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8.  AMO  F-16  Conversion  Training.  ANG  squadrons  oonvsrting  to  tbs 
F-16  found  ths  AZT  advantagsous  as  a  familiarisation  dsvios  in 
squadrons  and  as  part  of  ths  conversion  course  syllabus  at  the 
sohoolhouse.  Some  of  the  AFRES  units  relinquished  their  AXTs  for 
use  by  ANG  units. 


LESSONS  LEARNED  FROM  THE  ANG  EXPERIENCE:  ELABORATION 
1.  ANG  User  Acceptance  and  Training  Evaluation  Results 

Both  students  and  instructors  accepted  and  used  the  AIT 
enthusiastically  as  an  addition  the  F-16  conversion  course.  Early 
limitations  in  prototype  capabilities  were  corrected  and  the  AIT 
was  successfully  integrated  into  the  course  syllabus. . 

Laboratory  observers  spent  considerable  time  observing  AIT 
training  activities  and  interviewing  instructors,  students,  and 
managers.  The  data  collected  from  these  activities  provided 
initial  ev4  lence  of  the  high  level  of  training  utility  and  the  high 
level  of  acceptance  of  the  AIT  by  pilots.  Observations  extended 
over  a  period  of  several  months  and  involved  several  conversion 
classes.  Specific  data  from  questionnaires  included  the  amounts 
and  types  of  training  accomplished  by  individuals  on  the  AIT,  use 
of  instructional  features  by  both  students  and  instructors, 
instructors'  procedures  and  techniques,  instructor  and  student 
acceptance  of  device  features  and  capabilities,  requirements  for 
software  and  hardware  revisions,  and  device  reliability.  Findings 
were  also  used  to  improve  device  capabilities. 

The  study  of  how  pilots  used  the  instructional  features  of  the 
device  revealed  that  most  instructional  features  such  as  the  freeze 
and  scenario  replay  were  valuable  training  capabilities.  The 
instructional  capabilities  and  training  scenarios  were  accessible 
through  menu  options  which  pilots  found  easy  to  use. 

The  graphic  representation  modes  proved  to  be  particularly 
valuable.  As  visual  representations  of  the  intercept  designed  to 
supplement  corresponding  information  on  HUD  and  radar  displays, 
these  graphics  were  effective  in  aiding  pilots  to  understand 
intercept  geometry.  Instructors  reported  they  were  very  valuable 
in  aiding  students  to  better  "visualize"  the  spatial-temporal 
aspects  of  intercepts.  These  instructional  displays  required  few 
refinements  and  continued  to  be  one  of  the  most  effective  aspects 
of  AIT  training. 

The  second  part  of  the  evaluation  of  AIT  training  at  the  ANG 
involved  the  participation  of  student  pilots  in  an  experiment  to 
determine  the  effectiveness  of  transfer  of  training  from  the  AIT  to 
an  F-16  simulator.  The  objective  was  to  determine  if  training  in 
the  AIT  would  provide  an  advantage  to  students  entering  the 
simulator  phase  of  conversion  training.  Several  groups  of  student 
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pilots  were  trained  using  variations  of  AIT  training.  One  group  of 
pilots  received  no  AIT  training  and  served  as  a  control  group.  A 
computer-algorithm  was  used  to  score  pilot  performance  on  a  series 
of  intercepts  flown  in  the  simulator  as  a  transfer  of  training 
test.  All  pilots  who  participated  received  the  same  simulator 
performance  test. 

Results  showed  that  pilots  who  received  the  highest  amount  of 
AIT  training  tended  to  achieve  higher  scores  in  the  simulator. 
However,  methodological  problems  with  this  study  preclude 
conclusive  statements  based  upon  the  obtained  data.  Problems 
included  the  relatively  small  number  of  pilots  assigned  to 
experimental  groups,  the  large  number  of  instructors  used  in  the 
experimental  training  and  the  method  of  their  involvement,  the 
method  of  scoring,  group  differences  contributing  to  a  significant 
class  by  group  interaction,  and  questionable  use  of  pilots  with 
knowledge  of  intercept  as  subject  participants  in  the  study.  A 
complete  report  of  this  study  is  contained  in  Lahey,  Hubbard,  and 
Edwards  (1987) . 

However,  shortcomings  of  the  empirical  study  aside,  the 
evaluation  effort  did  permit  the  observation  of  the  practical 
training  benefits  from  the  AIT  for  the  ANG.  Opinions  from  both 
students  and  instructors  clearly  indicated  a  high  rate  of  device 
usage  and  acceptance  in  the  squadron. 

Useful  information  accrued  during  the  installation  and 
"shakedown"  of  the  AIT  at  the  ANG.  Working  through  the  training 
process  with  users,  observing  usage  patterns,  sampling  pilot 
opinion,  and  observing  training  methods,  all  contributed  to  the 
laboratory's  experience  and  to  technology  transition. 
Collectively,  these  experiences  were  valuable  for  purposes  of 
refining  the  laboratory  training  device  and  improving  the 
evaluation  process. 


2.  Familiarization  Training. 

The  development  of  effective  training  for  student  pilots  was  to 
some  extent  a  collaboration  between  instructors  at  the  162nd  TS  and 
software  developers  at  AL/HRA.  Despite  all  of  the  attempts  to  make 
the  AIT  operational  manual  easy  to  use,  it  was  found  that  most 
students  needed  time  to  accommodate  to  the  trainer.  This  involved 
becoming  comfortable  with  the  initialization  procedures  and  gaining 
facility  with  instructional  features  and  menus.  In  addition,  many 
students  found  that  HUD  and  radar  symbology  interpretation  skill 
required  more  time  than  had  been  anticipated.  This  aspect  of 
training  is  discussed  in  detail  in  #3  below. 
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An  early  assumption  by  those  developing  the  F-16  training 
syllabus  at  the  162nd  TS  was  that  student  pilots  would  be  able  to 
learn  basic  radar  interpretation  and  switchology  skills  during 
academics  prior  to  spending  time  on  the  AIT.  This  assumption 
included  the  belief  that  with  a  suitable  level  of  these  basic 
skills,  students  would  be  able  to  use  the  AIT  to  integrate  the 
switchology  with  intercept  procedures  and  then  devote  considerable 
time  to  practicing  whole  intercepts  using  the  various  scenarios 
available.  Training  time  allocations  in  the  initial  version  of  the 
syllabus  reflected  these  assumptions. 

Experience  with  the  first  several  classes  indicated  otherwise. 
Academics  training  including  media  materials  provided  insufficient 
training  to  produce  transferrable  hands-on  stick  and  throttle 
skills.  As  a  result,  instructors  found  students  spending  large 
amounts  of  their  AIT  time  working  on  basic  skills.  This  left 
inadequate  time  to  integrate  skills  and  practice  intercepts. 
Syllabus  changes  were  required  to  provide  additional  AIT  training. 

Instructors  were  persistent  in  the  view  that  the  AIT  should  be 
used  predominantly  for  whole-task  intercept  practice,  rather  than 
for  the  subskills.  As  a  result,  another  training  program  was 
developed  which  students  could  operate  off-line  on  a  personal 
computer  (Lahey  &  Anselm,  1989) .  This  program ,  which  became  known 
as  the  HUD/REO  Trainer,  provided  a  means  of  systematically 
assessing  pilot  readiness  for  AIT  training  and  thus  increased 
training  efficiency.  The  HUD/REO  became  a  standard  part  of  the 
revised  conversion  course  syllabus. 

4 .  Lack  of  Visual  Display. 

From  a  strict  training  perspective,  the  lack  of  the  visual 
display  proved  beneficial  because  pilots  had  to  rely  on  cockpit 
information  in  order  to  "fly"  the  simulated  aircraft.  In  this 
regard,  the  AIT  was  not  intended  to  teach  pilots  F-16  handling 
qualities.  In  the  final  analysis,  the  lack  of  the  visual  display 
turned  ou*  to  be  more  a  device  acceptance  issue  rather  than  a 
training  effectiveness  issue. 

As  a  postscript,  within  two  years  following  introduction  of  the 
AIT,  the  lack  of  the  outside  visual  scene  was  corrected  when  a 
dramatic  price  reduction  of  graphics  work  stations  made  the 
addition  of  the  display  an  affordable  upgrade  for  the  trainer. 
AITs  produced  following  the  price  reduction  were  equipped  with  an 
"outside”  visual  display  capability.  Most  existing  AITs  were  also 
retrofitted. 
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The  original  prototype  was  not  equipped  with  real-time,  fully 
pilot-controllable  flight.  The  pilot  was  able  to  control  only 
eegmented  portions  of  intercepts  interspersed  with  "freeze"  of 
flight  for  instructional  purposes.  In  the  initial  trainer  a 
computerized  model  of  the  correct  intercept  controlled  the  trainers 
flight  following  radar  lock-on,  so  that  the  pilot  was,  in  effect, 
"along  for  the  ride"  only.  A  fully  "flyable"  version  of  trainer 
software  was  under  development,  but  was  not  available  for  the  first 
several  conversion  course  classes.  Some  pilots  who  participated  in 
the  initial  training  effectiveness  evaluation  did  receive  some 
training  using  the  fully  flyable  software  model.  In  response  to 
pilot  perceptions  of  this  limitation,  AL/HRA  accelerated  work  on 
the  flyable  software  and  the  trainer  was  upgraded  within  about 
seven  months  after  the  initial  receipt  of  the  trainer  at  the 
squadron. 

Following  the  flyable  model  upgrade,  pilots  reported 
deficiencies  in  the  aerodynamics  and  engine  models  software.  The 
chief  complaint  about  aerodynamics  was  a  perceived  distortion  of 
the  elapsed  time  of  intercepts.  Many  indicated  that  the  time 
required  in  the  AIT  to  complete  an  intercept  seemed  less  than  that 
required  in  the  real  aircraft  for  the  same  intercept.  Another 
frequent  complaint  was  the  perception  that  the  trainer  stick 
control  was  too  sensitive  relative  to  the  aircraft,  resulting  in  a 
tendency  to  overcontrol  pitch  and  roll  during  intercept  training. 

Part  of  this  lesson  learned  resulted  from  the  required 
improvements  in  AIT  fidelity.  In  a  part-task  device,  fidelity  of 
essential  training  capabilities  is  paramount .  If  a  capability  is 
required  for  training,  high  fidelity  simulation  of  that  function  is 
essential.  Project  engineers  came  to  refer  to  this  concept  as 
selective  fidelity.  Any  function  simulated  must  be  functionally 
equivalent  to  the  aircraft.  Inexperienced  pilots  may  be  less 
critical  of  fidelity  compared  to  experienced  pilots,  but  unmet 
pilot  expectations  can  jeopardize  device  acceptance  and  project 
success.  Inadequate  fidelity  raises  the  fear  of  negative  training 
and  as  such,  is  always  a  concern. 

The  fidelity  of  the  AIT  prototype  was  deficient  in  several 
critical  areas  and  required  immediate  software  upgrades  to  meet 
training  requirements.  However,  from  the  outset,  pilot 
expectations  of  the  AIT's  training  capabilities  were  tempered  by 
the  fact  that  the  appearance  of  the  AIT  was  quite  abstract  (it  did 
not  look  like  a  cockpit) .  Once  the  needed  upgrades  were  installed, 
training  utility  and  pilot  acceptance  of  the  AIT  in  the  squadron 
were  essentially  achieved. 


6.  Intercept  Performance  Measurement. 


BttgKqrgynd _ on  the _ Original  Scoring  Algorithm.  Intercept 

performance  in  the  original  linear  model  was  measured  against  a 
parametric  model  of  an  idealized  intercept.  Flight  data  during 
each  intercept  attempted  by  the  pilot  were  captured  which  included 
30  measures  of  conformance  to  idealized  intercept  flight 
parameters.  Scores  assigned  for  each  measure  were  summed  to 
provide  a  single  numerical  score  for  the  intercept.  The  pilot 
could  earn  a  maximum  of  1,000  points  per  intercept.  Penalties  for 
errors  such  as  out-maneuvering  or  "gimballing"  the  radar,  breaking 
radar  lock,  negative  overtake  during  conversion,  failure  to  uncage 
the  missile  prior  to  launch,  and  excessive  time  to  complete  the 
intercept  resulted  in  points  subtracted  from  the  total  score. 
Thus,  negative  scores  on  intercepts  were  possible  and  were  in  fact 
observed  in  several  instances. 

The  numerical  weighting  of  intercept  components  was  found  to  be 
problematic  for  several  reasons:  (a)  experts  may  not  agree  on  what 
constitutes  the  "best"  intercept,  making  consensus  on  scoring 
difficult;  (b)  standards  of  performance  may  vary  across  squadrons 
based  upon  such  factors  as  rules  of  engagement  (ROE)  and  directed 
operational  capability  (DOC) ;  (c)  numerical  scores  alone  may  not 
provide  sufficiently  specific  feedback  to  help  pilots  improve 
performance;  and  (d)  pilots  may  resist  being  scored  by  computer. 

In  the  present  case,  the  rationale  for  selecting  and  weighting 
performance  components  was  not  subjected  to  sufficient  review  by 
experts.  This  resulted  in  the  assignment  of  inordinate  weights  to 
some  factors.  For  example,  the  penalty  for  gimballing  the  radar 
was  minus  400  points  from  the  1,000  point  total.  This  turned  out 
to  be  too  heavy  a  penalty  for  this  factor  and  resulted  in  severely 
skewed  scores  for  a  number  of  pilots. 

Results  from  Findings.  This  lesson  learned  combined  with  a 
number  of  years  experience  in  dealing  with  measurement  of  complex 
pilot  performance  led  to  concerted  efforts  to  develop  feedback 
techniques  to  aid  pilots  obtain  accurate  representations  of 
performance.  This  more  recent  approach  is  based  on  the  assumption 
that  if  pilots  are  provided  sufficiently  clear  and  complete 
feedback  on  their  performance,  they  are  likely  to  identify  mistakes 
and  make  corrections  themselves,  or  in  consultation  with  experts. 
Exploiting  computer  capabilities  to  support  this  approach  for  air 
intercept  training  has  been  reasonably  successful  in  the  case  of 
the  AIT. 

Enhanced  debriefing  capabilities  as  AIT  training  features  have 
been  developed  and  used  successfully. 
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7.  Training  Effectiveness;  Empirical  Data. 

Obtaining  reliable  evidence  of  the  training  effectiveness  of 
the  AIT  required  concerted  effort  to  ensure  a  suitable  experimental 
design  methodology  for  field  testing  of  the  device.  While  the 
initial  training  effectiveness  evaluation  produced  inconclusive 
data,  lessons  learned  from  that  effort  were  applied  to  a  subsequent 
device  training  evaluation.  Data  collection  for  the  second 
evaluation  required  more  than  a  year,  but  resulted  in  clear 
evidence  of  the  AIT's  training  utility  and  effectiveness. 

8.  Prototyping  Methodology . 

The  approach  taken  by  AL/HRA  was  to  get  ahead  of  the  "game"  by 
designing,  as  it  were,  with  "future"  technology.  The  VME-1000,  a 
68020-based  system,  was  selected.  At  the  time,  this  microcomputer 
design  was  so  new  it  was  still  in  an  unrefined  state  from  the 
manufacturer.  This  system  was  selected  largely  on  the  merits  of 
its  potential  to  accommodate  long-term  future  growth  and 
flexibility  for  configuration  changes.  The  penalty  paid  was  in 
working  through  some  system  flaws  and  needed  refinements  jointly 
with  the  manufacturer. 

What  evolved  from  this  lesson  learned  was  a  two-phase 
prototyping  process.  A  first-stage  device  was  used  as  an 
engineering  test-bed  for  smoothing  the  "ragged  edge"  of  "green" 
technology.  A  second  prototype  was  used  for  the  actual  product 
development.  This  made  short-term  "freezes"  of  the  trainer  design 
acceptable  to  engineers  who  could  rely  on  the  first-stage  device. 
Knowledge  gained  from  experimentation  using  the  first-stage  device 
gave  engineers  a  flexible  tool  to  determine  when  new  technology  was 
ready  for  insertion  into  the  (second-stage)  product.  The  process 
outcome  was  as  intended:  the  user  received  the  latest  technology  on 
a  timely  basis. 

Another  aspect  of  the  prototyping  process  involved  SMEs.  As 
the  AIT  program  grew,  users  multiplied.  Rather  than  going  to 
various  user  squadrons  to  test  changes  and  correct  system  problems, 
it  became  more  efficient  to  invite  pilots  from  several  units  to 
AL/HRA  to  test  innovations.  Using  this  approach,  it  was  sometimes 
possible  to  eliminate  problems  before  exporting  the  changes  to 
users.  This  aspect  of  the  prototyping  lesson  learned  became  an 
engineering  benchmark  and  is  now  standard  practice  for  training 
device  development  at  AL/HRA. 


LESSONS  LEARNED  FROM  THE  ACC  SCHOOLHOUSE  EXPERIENCE:  ELABORATION 

As  mentioned  earlier,  the  ACC  schoolhouse  trainer  contained 
several  enhancements  beyond  the  original  prototype.  As  in  the 
original  trainer,  menu  access  to  training  scenarios  provided 
students  with  ordered  levels  of  task  complexity  and  challenge.  In 
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the  upgraded  schoolhouse  trainer,  the  type  and  number  of  intercept 
scenarios  was  expanded  to  support  training  in  the  basic  areas  of 
radar  employment,  multiple  target  sorting,  and  intercept  geometry. 
The  enhanced  scenarios  provided  the  instructor  options  to 
manipulate  the  number  and  configuration  of  radar  targets  via  the 
student  instructor  control  station  (SICS) . 

1.  Utility  and  Training  Effectiveness 

As  it  had  been  with  the  original  AIT,  the  AL/HRA  intent  during 
the  ACC  schoolhouse  phase  was  to  make  a  systematic  study  of  device 
acceptance  and  training  utility.  During  the  schoolhouse  phase, 
questionnaires,  interviews,  and  a  formal  training  experiment  were 
conducted  in  the  squadron  to  collect  evidence  of  device  acceptance, 
utility,  and  training  effectiveness. 

Student  Opinion.  Two  questionnaires  were  distributed  to 
students  enrolled  in  the  B-Course.  Students  were  asked  a  variety 
of  questions  about  the  capabilities  of  the  AIT  and  their  use  of  the 
device.  A  third  questionnaire  was  circulated  to  students  enrolled 
in  the  B-course  at  the  56th  Training  Squadron  at  MacDill  AFB,  FL 
following  the  delivery  of  an  AIT  for  that  ACC  schoolhouse,  about 
two  years  following  the  advent  of  the  AIT  at  the  Luke  squadron.  A 
summary  of  student  pilot  opinion  from  these  questionnaires, 
relative  to  trainer  effectiveness  follows. 

a.  AIT  acceptance  and  training  effectiveness  at  the 
schoolhouse  was  based  upon  lessons  learned  from  the  ANG  experience. 
A  considerable  number  of  additional  capabilities  and  refinements 
were  completed  by  the  time  the  AIT  arrived  for  evaluation  at  the 
ACC  schoolhouse.  With  few  exceptions,  the  instructional  features 
and  training  capabilities  developed  at  the  ANG  served  well  in  the 
B-course . 


b.  All  the  training  features  in  the  AIT  were  employed  by 
B-course  students,  some  more  than  others.  However,  none  of  the 
training  features  were  used  by  less  than  25%  of  students.  The 
soundness  of  the  training  analysis  and  design  underlying  these 
features  was  confirmed  by  the  response  of  students.  Of  all  the 
features  embodied  in  the  trainer,  perhaps  the  most  successful, 
judging  by  usage  data,  were  the  ancillary  graphics  displays  on  the 
SICS  used  to  convey  intercept  geometry. 

In  responding  to  questionnaires,  pilots  indicated  the  AIT  aided 
their  development  of  skills  in  the  following  specific  areas: 
general  radar  employment,  target  sorting  and  management,  intercept 
geometry,  situational  awareness,  and  overall  execution  of 
intercepts . 

Students  reported  the  AIT  provided  an  effective  "bridge" 
between  classroom  instruction  and  "hands-on"  system  training. 
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Students  indicated  they  found  instructional  capabilities  reasonably 
"user  friendly." 

c.  AIT  intercept  training,  as  an  adjunct  to  the 
conventional  syllabus ,  was  relevant  and  timely,  and  highly 
accepted.  Students  responded  to  the  AIT  as  an  opportunity  to 
operationalize  concepts  introduced  in  academics  classes.  They 
reported  that  what  they  had  learned  in  class  became  immediately 
relevant  via  the  AIT.  The  convenience  of  having  the  AIT  present  in 
the  squadron  facilitated  learning. 


d.  In  responding  the  questionnaire  items  related  to  trainer 
fidelity,  pilots  indicated  general  agreement  that  basic  avionics 
functions  of  the  trainer  (master  radar  modes,  weapons  switchology, 
HUD  functions,  and  overall  trainer  performance)  performed  well 
referenced  to  the  aircraft.  There  was  less  agreement  about  the 
accuracy  of  flight  controls.  Pilots'  reservations  about  the 
control  sensitivity  seemed  to  carry-over  from  those  of  pilots  in 
the  ANG.  But  the  control  sensitivity  shortcoming  was  not  shown  to 
be  detrimental  to  device  training  effectiveness.  Both  students 
and  instructors  would  have  appreciated  improved  control  fidelity  as 
well  as  the  availability  of  targets  with  more  sophisticated 
capabilities.  But  these  improvements  were  beyond  the  level  of 
fidelity  required  to  support  syllabus  requirements  and  would  have 
been  too  expensive  to  justify  their  addition. 

e.  While  the  essential  design  of  training  software  was 
effective,  considerable  time  was  required  for  pilots  to  learn  the 
basic  operation  of  the  device.  A  number  of  pilots  had  trouble 
getting  the  trainer  up  and  operating.  This  problem  had  not  been 
fully  resolved  by  the  end  of  the  period  of  schoolhouse  evaluation. 

Training  Effectiveness  Evaluation.  A  formal  training 
experiment,  by  AL/HRA,  was  conducted  using  B-course  students  as 
subjects  to  test  the  training  effectiveness  of  the  AIT.  Lessons 
learned  from  an  earlier  training  effectiveness  evaluation, 
conducted  at  ANG,  Tucson  were  applied  to  improve  experimental 
methodology.  Fifty  student  pilots  across  several  B-Course  classes 
participated.  The  experiment  involved  two  groups  of  students,  25 
in  each  group.  As  a  part  of  academics  training  on  intercepts,  one 
group  received  training  on  the  AIT,  the  other  group  received  no  AIT 
training.  Following  the  experimental  training,  all  students  in 
both  groups  were  tested  on  their  ability  to  perform  air  intercepts 
in  the  operational  flight  trainer  (F-16  simulator)  as  a  means  to 
test  transfer  effects. 

Results  from  this  experiment  are  summarized  as  follows: 
Relative  to  the  non-AIT  trained  group,  the  AIT-trained  group  (a) 
achieved  higher  proficiency  ratings  in  four  or  five  basic  skill 
areas,  (b)  achieved  performance  proficiency  in  significantly 
greater  numbers  and  on  significantly  more  types  of  intercepts,  (c) 
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were  able  to  fly  significantly  more  advanced  intercepts  during  the 
simulator  training,  and  (d)  on  advanced  intercepts,  a  significantly 
higher  percentage  of  AIT-trained  pilots  reached  acceptable  levels 
of  proficiency.  A  complete  report  of  this  study  is  contained  in 
Edwards  and  Hubbard,  1991. 

Thus,  data  from  both  pilots  opinion  surveys  and  the  formal 
training  experiment  clearly  substantiated  the  utility  and 
effectiveness  of  AIT  training  in  the  B-course. 

2.  Self-Paced  Training  Capability.  Provision  for  self-paced 
training  was  a  design  issue  during  development  of  the  AIT. 
Typically,  simulators  require  the  presence  of  an  instructor  during 
training  activities.  The  AIT,  although  a  part-task  trainer,  was 
partially  an  aircraft  simulator.  Training  capabilities  were  an 
integral  part  of  trainer  software.  The  design  issue  was  one  of 
training  efficiency.  If  the  device  could  be  used  successfully  in 
the  self-paced  mode  by  the  student,  substantial  instructor  time 
could  be  saved  during  the  course.  Related  questions  were  raised. 
Would  students  be  motivated  to  use  the  trainer  by  themselves? 
Would  training  be  effective  without  an  instructor?  Would 
instructors  view  self-pacing  as  a  help  or  hindrance  in  the  course? 

The  results  of  evaluation  showed  that  after  sufficient 
familiarization  with  the  AIT,  most  students  came  to  prefer  the 
self-paced  mode  training.  They  were  able  to  use  the  training  menus 
and  features  independently  of  the  instructor.  The  most  frequently 
mentioned  comment  from  students  was  that  they  liked  the  self-paced 
mode  because  of  the  independence  it  afforded  them.  Typical 
comments  were,  "There  are  a  lot  of  things  you  need  to  learn  on  your 
own  in  the  AIT,"  and  "We  need  to  do  it  our  own  way;  that  works 
best."  Moreover,  students  took  advantage  of  the  presence  of  the 
trainer  in  the  squadron  which  added  convenience  and  flexibility  to 
training  schedules. 

Relative  to  instructors ,  when  they  became  convinced  of  the 
capabilities  of  the  AIT,  and  with  the  ability  of  students  to  use 
the  device  alone  successfully,  there  were  no  reservations  about 
self-paced  usage.  Indeed,  instructors  came  to  regard  the  AIT  as  a 
valuable  asset  which  allowed  them  to  reapportion  instructional  time 
to  other  training  areas.  However,  this  acceptance  came  only  after 
instructors  had  personally  observed  and  evaluated  student  use  of 
the  AIT  and  "reconciled"  the  "solo"  capability  with  the  existing 
"ways  of  doing  things"  at  the  schoolhouse. 

3.  Instructor  Acceptance.  During  the  introduction  of  the  AIT  at 
the  schoolhouse,  instructors  from  both  academics  and  the  flight 
line  used  the  AIT  and  participated  in  its  evaluation.  The 
reactions  of  instructors  were  viewed  as  critical  to  the  successful 
implementation  of  the  AIT  in  the  squadron.  For  academics 
instructors,  understanding  of  the  AIT  was  more  extensive  because 
most  of  them  spent  considerable  time  with  students  on  the  device. 
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A  few  of  the  instructors  had  also  been  consulted  by  AL/HRA  during 
the  design  of  AIT  training  software.  Instructors  were  interviewed 
about  AIT  capabilities  and  how  they  used  it  for  training. 

At  first,  the  general  attitude  of  instructors  toward  the 
trainer  was  one  of  mild  skepticism.  Several  other  part-task 
devices  had  preceded  the  AIT  in  the  squadron.  These  trainers  had 
been  only  "tolerated"  because  they  added  some  training  value,  but 
left  much  to  be  desired  as  effective  and  reliable  training 
equipment.  So,  for  the  AIT,  instructors  seemed  to  adopt  a  "wait 
and  see"  attitude.  However,  by  the  end  of  the  training  evaluation 
period,  collective  instructor  opinion  was  highly  favorable,  in 
spite  of  several  areas  of  needed  improvement  identified  by 
instructors.  A  summary  of  instructor  opinion  about  the  AIT, 
compiled  from  interview  data,  appears  below. 

Training  Capability.  Probably  the  most  often-mentioned 
training  benefit  identified  by  instructors  was  that  it  provided 
students  with  what  is  sometimes  referred  to  as  the  "big  picture." 
All  instructors  interviewed  perceived  the  AIT  as  aiding  students  to 
develop  a  facility  to  "move"  mentally  between  two-dimensional 
information  (e.g.,  cockpit  displays)  and  a  concept  of  relationships 
between  two  or  more  aircraft  maneuvering  in  the  real  (three- 
dimensional)  environment.  Instructors  indicated  that  a  grasp  of 
this  dynamic  "geometry"  is  perhaps  the  single  most  important 
concept  during  the  air-to-air  phase  of  the  course .  Instructors 
noted  that  students  using  the  AIT  seemed  to  grasp  spatial/temporal 
concepts  more  quickly  than  non-AIT  trained  students. 

Instructional  Content.  There  was  general  agreement  among 
instructors  that  the  content  and  structure  of  training  scenarios 
was  appropriate  for  the  B-Course.  This  acceptance  was  related  to 
the  method  by  which  scenarios  were  designed  by  AL/KRA. 
Conceptualization  and  development  of  scenarios  was  accomplished  by 
instructional  designers  and  involved  extensive  consultation  with 
subject-matter  experts  over  an  extended  time  period.  In  several 
cases,  instructors  from  the  F-16  training  community  had  consulted 
with  the  designer  on  instructional  content.  Some  instructors  were 
also  aware  of  the  implementation  of  the  original  device  at  Tucson. 
Thus  by  the  time  the  trainer  arrived  on  site  at  the  Luke  AFB 
schoolhouse,  its  instructional  capabilities  had  been  subjected  to 
critique  by  various  pilots.  One  aspect  of  this  lesson  learned, 
therefore,  underscored  the  importance  of  involving  expert  users 
early  in  device  design  and  continuing  consultation  during 
evaluation. 

Availability  of  Practice  Time.  Instructors  regarded  the 
capability  to  practice  intercepts  afforded  by  the  AIT  as  very 
important  during  academics.  The  B-course  syllabus  provided  only 
three  simulator  sorties  and  three  aircraft  sorties  which  contained 
substantial  time  devoted  to  air  intercepts.  By  contrast,  the  AIT 
afforded  a  relative  abundance  of  time  to  students  to  work  out  an 
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understanding  of  intercepts  and  to  acquire  transferrable  skills  for 
simulator  and  aircraft  training.  In  this  regard,  instructors  also 
said  they  found  the  fidelity  of  the  AIT  sufficiently  accurate  for 
training  transfer.  They  regarded  fidelity  as  critical  to  skill 
transfer  in  that  when  students  worked  out  an  idea  or  technique 
using  the  AIT,  they  could  be  confident  that  it  would  work  the  sane 
way  in  the  aircraft. 

A  goal  of  AIT  design  was  available,  affordable  training  in  the 
squadron  for  specific  basic  skills.  This  goal  was  successfully 
met.  Class  size  averaged  13  or  14  students  during  the  evaluation 
period.  Student  scheduling  of  the  trainer  turned  out  to  be 
workable,  in  spite  of  a  few  "bottlenecks. M  Instructors  noted  that 
some  of  the  more  motivated  students  would  come  in  on  weekends  to 
use  the  AIT.  The  availability  of  the  device  seven  days  per  week 
helped  because  it  provided  maximum  flexibility  for  students  to 
arrange  schedules.  While  the  AIT  did  add  training  time  to  the 
syllabus,  its  accessibility  to  students  in  the  squadron  helped 
offset  the  need  for  additional  training  time. 

Skill  Integration.  Another  prominently-mentioned  training 
quality  was  the  capability  to  "work  different  skills  together"  or 
to  integrate  skills.  One  example  is  gaining  facility  with  throttle 
and  stick  switches  when  identifying  and  sorting  multiple  targets  on 
the  radar  screen.  Instructors  viewed  this  aspect  of  training  as 
being  a  unique  advantage  afforded  by  the  AIT,  and  one  particularly 
useful  in  preparing  students  to  spend  time  productively  in  the 
simulator. 

Needed  Improvements .  All  of  the  instructors  indicated  they 
experienced  some  difficulty  learning  to  control  flight  on  the  AIT. 
The  stick  seemed  oversensitive  causing  overcontrol  of  pitch  and 
roll  inputs.  Some  instructors  found  that  the  trainer  was  not  able 
to  accelerate  like  the  aircraft,  another  aspect  of  flight  fidelity. 
They  also  found  need  for  improvement  with  the  target  capabilities 
of  various  intercept  scenarios.  Most  indicated  targets  were  too 
slow  generally  and  some  said  targets  should  be  able  to  maneuver,  at 
least  to  some  degree.  Several  instructors  said  the  pilot  should  be 
more  easily  able  to  "get  out"  of  (disengage)  the  multiship 
intercept  scenarios.  These  findings  were  used  as  inputs  to  improve 
the  fidelity  and  functional  capabilities  of  the  AIT  in  the 
schoolhouse. 

4.  Adoption  of  the  Trainer.  Following  the  initial  positive 
evaluation  of  the  AIT  by  instructors  and  students  at  the  Luke 
schoolhouse,  the  hope  of  the  4444th  Operations  Squadron,  Detachment 
1  was  that  its  formal  adoption  into  the  B-Course  syllabus  would  be 
expeditious.  However,  the  syllabus  could  not  be  formally  revised 
until  the  other  F-16  schoolhouse  at  MacDill  AFB  (56th  TS)  also 
received  an  AIT. 
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ACC  had  accepted  the  AIT  as  a  testbed  device  by  memorandum  of 
agreement  as  a  quid  pro  quo  for  AL/HRA  research  privileges  at  the 
schoolhouse.  No  funds  were  solicited  nor  received  by  AL/HRA  from 
ACC  for  the  trainer.  The  response  to  the  trainer  at  Luke  had  been 
very  favorable  and  subsequent  empirical  data  from  the  formal 
training  effectiveness  evaluation  clearly  demonstrated  benefits  of 
the  device  in  the  B-course.  However,  this  evidence  appeared  to  be 
misunderstood  or  ignored  by  Headquarters  ACC/DOT.  While  the 
laboratory  continued  to  support  the  O&M  costs  of  the  AIT  in  the 
schoolhouse  at  no  cost  to  ACC,  the  reasonable  expectation  was  that 
a  second  device  would  be  provided  for  the  HacOill  schoolhouse  in  a 
timely  manner.  However,  HQ  ACC  questioned  the  presence  of  the  AIT 
in  the  schoolhouse.  Two  issues  were  raised  by  ACC/DOT:  (a)  the 
need  for  a  specialized  trainer  for  air-to-air  training  was 
questioned  because  the  F-16  role  was  primarily  air-to-ground, 
implying  that  emphasis  on  training  resources  should  be  deferred  to 
the  latter  role;  and  (b)  since  the  schoolhouse  had  successfully  met 
training  requirements  before  the  AIT,  its  addition  to  the  B-course 
was  unjustified  by  requirements.  Notwithstanding  the  terms  of  the 
memorandum  of  agreement  between  ACC  and  AL/HRA,  HQ  ACC  seemed  to 
regard  the  AIT  as  an  unsolicited  addition  to  the  course.  Thus,  the 
logical  inference  from  these  objections  was  that  HQ  ACC  saw  the  AIT 
merely  as  a  "nice-to-have1'  device.  Fortunately,  Training  Systems 
Center  at  Luke  AFB  had  begun  to  produce  AITs  in  quantity  and  was 
able  to  acquire  sufficient  spare  hardware  components  to  assemble  a 
second  schoolhouse  device.  Because  the  costs  of  trainer  software 
development  had  been  paid  by  AL/HRA,  the  cost  of  the  second  device 
was  only  for  hardware.  At  length,  4444th,  Det  1  was  able  to 
convince  headquarters  that  the  training  value  of  the  AIT  was, 
indeed,  worth  the  cost  of  the  hardware.  The  second  device  was 
assembled  and  delivered  to  the  MacDill  AFB  schoolhouse  in  late 
1991,  some  two  years  after  the  completion  of  the  training 
effectiveness  evaluation  at  Luke. 


LESSONS  LEARNED  FROM  THE  AFRES  EXPERIENCE:  ELABORATION 

With  the  notable  success  of  the  AIT  in  the  schoolhouse, 
consideration  of  extended  applications  seemed  logical.  One 
possibility  was  modification  of  the  trainer  for  operational  users, 
particularly  the  Air  Force  Reserve  units  flying  F-16  aircraft.  The 
only  ground-based  trainers  available  in  the  Reserve  during  the  late 
1980s  were  several  older  F-16A  simulators  which  were  regarded  of 
marginal  value  because  of  limited  visual  capabilities,  poor 
reliability,  high  operations  and  maintenance  (O&M)  costs  and 
limited  availability  for  use  by  reservists.  These  factors  led  to 
a  decision  by  the  Reserve  to  phase  out  the  trainers.  By  1990,  for 
example,  the  466th  Fighter  Squadron  (FS) ,  Hill  AFB,  UT  no  longer 
had  a  simulator  available.  The  302nd  FS,  Luke  AFB  had  only  limited 
access  to  a  simulator,  the  F-16  operational  flight  trainer  (OFT) , 
because  of  the  priority  for  that  device  in  the  F-16  schoolhouse. 
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The  primary  AFRES  interest  in  ground-based  training  equipment 
was  in  three  areas  considered  critical  to  maintenance  of  safety  of 
flight  standards,  particularly  for  reservist  pilots:  (1)  emergency 
procedures,  (2)  instrument  landings,  and  (3)  air-to-air  operations. 
Skill  maintenance  was  of  considerable  interest  because  of  the  large 
number  of  pilots  in  reserve  units  who  spend  frequent,  extended 
periods  of  time  away  from  the  F-16.  Although  the  AIT  offered 
training  capabilities  for  only  limited  aspects  of  air-to-air 
training,  it  was  viewed  by  Reserve  management  as  promising 
technology  with  several  virtues:  (a)  low  cost,  (b)  timely 
availability  compared  to  conventional  acquisition,  and  (c)  design 
flexibility  with  expansion  potential  for  other  training  functions. 

The  AFRES  decision  to  pursue  AIT  enhancement  was  a  technology 
investment  strategy  approach.  As  a  means  of  providing  a  first  look 
at  training  feasibility,  the  302nd  FS  at  Luke  AFB  was  selected  to 
try  out  the  AIT.  AL/HRA  had  been  developing  the  ability  to  network 
two  AITs  for  two-ship  training  for  about  a  year  when  the  first  two- 
ship  AIT  configuration  was  delivered  to  the  302nd  for  evaluation. 
Training  software  in  the  device  was  approximately  the  same  as 
contained  in  the  schoolhouse  version.  Operational  pilots 
experimented  with  the  device  for  about  four  months  during  the 
latter  half  of  1988. 

During  this  trial  period,  it  became  clear  that  the  software 
setups  as  designed  for  the  schoolhouse  were  largely  inappropriate 
for  use  by  operational  pilots.  The  experienced  pilots  were  well 
beyond  being  able  to  benefit  from  basic  intercept  maneuver 
scenarios,  even  in  a  multiship  mode.  The  determination  of  how 
training  scenarios  should  be  upgraded  was  the  subject  of 
considerable  study  by  laboratory  personnel  in  consultation  with 
AFRES  subject  matter  experts.  Required  training  enhancements 
identified  were  as  follows: 

1.  Improved  adversary  engagement  scenarios  in  the  two-ship 
(networked  mode)  versus  1,2,4,  and  6  computer  controlled  targets. 

2.  Improved  fidelity  in  the  aerodynamic  performance  of  the 
trainer. 

3.  Improved  fidelity  of  radar  simulation  and  targeting. 

4.  Capability  to  engage  maneuvering  targets  with  a  higher 
degree  of  realism. 

The  organization  of  scenarios  for  the  schoolhouse  was  also 
found  to  be  cumbersome  for  operational  pilots.  They  wanted  quick 
access  to  advanced  complexity  scenarios  without  the  need  to 
"branch"  through  a  hierarchy  of  other  scenarios.  In  addition, 
AFRES  needed  scenarios  of  considerably  greater  combat  complexity, 
and  also  a  capability  for  the  squadron  weapons  system  officer  to 


manipulate  scenario  content  to  meet  various  situational 
requirements  for  unit  training  purposes. 

As  a  result  of  findings  from  the  trial  period,  AL/HRA 
developers  attempted  to  provide  enhancements  acceptable  to  AFRES. 
As  will  be  seen,  both  fidelity  and  training  scenario  enhancements 
proved  to  be  substantial  challenges. 

Although  the  identified  enhancements  were  still  under 
development,  AFRES  decided  to  pursue  an  expanded  use  of  the  AIT 
employing  the  hardware  assembly  capabilities  of  the  Training 
Systems  Center  at  Luke  AFB.  The  basic  arrangement  was  that  AL/HRA 
would  continue  to  develop  software  for  the  trainer  while  TSC  would 
assume  the  job  of  assembling  a  number  of  Alls  for  AFRES  and  other 
customers.  As  software  enhancements  were  completed,  they  would  be 
"exported"  to  device  users.  The  primary  interest  of  the  laboratory 
would  be  in  developing  effective  software;  the  primary  interest  of 
TSC  would  be  in  hardware  assembly  and  device  maintenance  for  field 
users.  Under  this  arrangement,  AFRES  began  deploying  AITs  to  F-16 
operational  units.  These  units  included  the  466th  FS,  Hill  AFB, 
UT;  the  89th  FS,  Wright-Patterson  AFB,  OH;  the  465th  FS,  Tinker 
AFB,  OK;  and  the  704th  FS,  Bergstrom  AFB,  TX.  AFRES  units  located 
at  Carswell  AFB,  TX  and  Homestead  AFB,  FL  used  the  AITs  for  short 
periods  of  time  but  did  not  retain  the  devices. 

During  the  period  of  deployment,  user  surveys  and  interviews 
with  pilots  were  conducted  by  AL/HRA.  These  data  collection 
activities  spanned  a  period  of  about  two  years  and  involved  the 
participation  of  53  pilots  who  responded  to  questionnaires  and 
interviews  at  various  units. 

Users  responded  to  questions  covering  a  variety  of  aspects  of 
training  using  the  AITs.  These  areas  included  the  type,  extent  and 
patterns  of  use,  preferred  modes  of  use,  training  features  and 
capabilities,  problems  encountered  while  using  the  device,  pilot 
assessment  of  various  aspects  of  device  fidelity,  perceived  need 
for  improvements,  perceived  needed  improvements  in  device 
management,  most  and  least  liked  aspects  of  AIT  training,  and 
general  comments  about  the  AIT. 

1.  Trainer  Utility  for  Operational  Pilots. 

Device  Usage .  Pilots  reported  using  the  AIT  an  average  of 
about  3.5  times  per  month.  Usage  patterns  varied  considerably 
across  units  and  by  pilot  experience  and  time  in  the  unit. 

Preferred  Mode  of  Use.  Pilots  indicated  that  85%  of  the  time 
they  used  the  AIT  alone,  rather  than  in  the  two-ship  mode  with 
other  pilots.  This  finding  was  disappointing  in  that  the  purpose 
of  developing  the  network  mode  was  to  permit  these  pilots  to 
simulate  two-ship  operations.  Interview  data  showed  that  pilots 
preferred  independent  use  of  the  device  largely  as  a  matter  of 
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convenience  and  also  because  many  seemed  to  feel  the  benefits  of 
the  two-ship  mode  were  marginal  compared  to  the  effort  required  to 
coordinate  schedules  of  other  pilots.  Another  problem  was  device 
reliability.  Some  pilots  candidly  stated  that  the  value  of  the 
two-ship  mode  did  not  justify  the  time  and  difficulty  getting  the 
system  to  work  reliably. 

Use  of  Scenarios.  Virtually  all  pilots  reported  using  the 
multiple  target  scenarios.  These  were  the  most  complex  available 
in  the  AIT  and  it  was  clear  that  for  operational  pilots,  these 
"high  end"  scenarios  were  the  only  ones  of  interest.  The 
operational  pilot,  as  it  turned  out,  challenged  the  capabilities  of 
the  AIT  to  the  utmost.  Many  pilots  indicated  that  they  needed  much 
more  capability  in  the  computerized  targets. 

Trainer  Fidelity.  Pilots  rated  the  fidelity  characteristics  of 
the  AIT  well  below  the  ratings  given  by  student  and  instructor 
pilots  in  the  schoolhouse. 

Needed  Improvements .  The  major  areas  of  needed  improvement 
indicated  by  pilots  (in  order  of  magnitude)  were: 

1.  Improved  aircraft  aerodynamics ,  controls,  and  avionics. 

2.  Improved  target  maneuvering  capabilities. 

3.  Improved  multiship  capabilities  and  operation. 

4.  Added  and  improved  weapons  capabilities. 

Training  Value  of  AIT.  Pilots  were  asked  to  rate  the  value  of 
AIT  training  in  the  worth  of  their  own  time  and  the  impact  of  AIT 
training  on  unit  readiness.  Responses  showed  that  while  pilots 
were  not  totally  negative  toward  the  AIT,  they  perceived  its  value 
to  be  limited  by  fidelity  and  training  capabilities. 

Check  Ride  Innovation.  Of  the  AFRES  squadron  using  the  AIT, 
the  89th  FS  seemed  to  derive  somewhat  more  training  value  from  the 
device.  This  was  primarily  because  the  majority  of  pilots  were 
relatively  new  to  the  F-16.  The  unit  training  officer  indicated 
they  used  the  AIT  to  augment  portions  of  the  annual  air-to-air 
tactics  check  ride  required  for  all  pilots.  The  requirement  for 
AIT  training  was  in  addition  to  the  aircraft  ride  itself.  Pilots 
used  the  AIT  to  sharpen  basic  skills  prior  to  the  check  ride. 
They  were  allowed  the  option  of  selecting  any  of  the  2v- 
maneuvering  multitarget  scenarios  for  their  check  ride  rehearsal. 
This  use  of  the  AIT  appeared  to  be  a  significant  training 
innovation  in  this  squadron. 

2.  Basic  Skill  Refresher  Value.  The  value  of  the  AIT  for  refresh 
training  among  less  experienced  reservist  pilots  was  best 
illustrated  at  the  89th  FS.  Many  of  these  pilots  who  had 


24 


experience  levels  of  from  80  to  150  hours  in  the  F-16  spent 
considerably  more  time  using  the  AIT  than  pilots  in  other  AFRES 
units.  Substantial  amounts  of  their  training  in  the  device  were 
devoted  to  basic  skill  refresh  training.  The  weapons  system 
officer  indicated  that  a  number  of  the  newer  pilots  came  to  the 
squadron  "on  their  own  time"  for  the  express  purpose  of  using  the 
\  AIT.  Questionnaire  data  also  showed  that  89th  FS  pilots  tended  to 

rate  the  value  of  AIT  training  considerably  higher  than  pilots  with 
more  experience  in  other  units.  in  contrast  to  this  unit,  pilot 
experience  at  the  466th  FS,  Hill  AFB  averaged  over  700  hours  per 
pilot.  Only  a  few  pilots  in  the  466th  had  fewer  than  200  hours  in 
the  aircraft.  The  use  of  the  AIT  for  basic  skill  refreshment 
accounted  for  only  a  minor  amount  of  training,  even  among  part-time 
reservists  at  the  466th.  Overall,  the  use  of  the  AIT  for 
switchology  refresh  was  viewed  by  the  AFRES  operational  squadrons 
as  a  relatively  minor  training  benefit. 

3.  Fidelity.  Pilots  in  the  AFRES  units  were  considerably  more 
critical  of  AIT  fidelity  than  those  in  the  schoolhouse.  There  were 
significant  and  continuing  complaints  about  over sensitivity  of 
flight  controls.  The  aerodynamics  and  engine  simulation  models  had 
been  checked  and  rechecked  for  accuracy  with  the  aircraft.  At 
length,  expert  opinion  favored  the  view  that  the  perception  of 
control  over sensitivity  was  due  to  a  combination  of  two  factors: 
(a)  the  difference  between  stick  "feel"  in  the  aircraft  and  that  of 
the  AIT,  and  (b)  the  absence  of  peripheral  visual  cuing  in  the 
trainer  because  of  its  limited  field-of-view  visual  display.  To 
simulate  the  effects  of  the  aircraft  stick  force  transducer  in  the 
AIT  would  have  added  greatly  to  the  cost  of  the  trainer.  The 
potential  training  value  of  this  enhancement  to  the  AIT  was  judged 
to  be  worth  much  less  than  its  high  cost,  so  it  was  not  added. 

4.  Design  Enhancements.  Simply  stated,  the  AIT  was  not  well 
utilized  in  the  AFRES  operational  squadrons.  Attempts  to  improve 
the  performance  of  the  trainer  in  order  to  meet  the  expectations  of 
AFRES  pilots  in  these  units  met  with  limited  success.  Except  for 
switchology  refreshment,  AFRES  pilots  were  not  deriving  substantial 
tactical  training  benefits  at  the  time  usage  surveys  were  taken  in 
the  various  units. 

Study  of  the  enhancement  potential  of  the  AIT  spanned  a  three- 
month  period  in  which  AL/HRA  interviewed  subject  matter  experts  to 
identify  the  specific  improvements  required  by  AFRES  pilots  to  make 
the  device  acceptable.  An  initial  list  of  capabilities  was 
developed  and  then  subjected  to  wider  review  among  AFRES  managers 
and  pilots.  The  endorsement  of  experts  developing  the  list  came 
with  the  statement,  "these  changes,  if  properly  implemented,  would 
eradicate  the  perception  of  the  AIT  as  a  "lieutenant  trainer"  by 
challenging  the  skills  and  abilities  of  senior  pilots."  The  list 
of  capabilities,  with  descriptions  condensed,  appears  below. 
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a.  Addition  of  the  Radar  Homing  and  Warning  (RHAW)  scope  to 
reflect  true  signatures  of  air-to-air  threats.  A  beyond-visual- 
range  (BVR) ,  air-to-air  threat  warning  would  be  required  because  of 
the  lack  of  ground  control  intercept  (GCI)  information  in  the  AIT. 

b.  Addition  of  "intelligent  adversarial  capabilities. *  This 
enhancement  is  subdivided  into  three  aspects:  (a)  maneuvering 
capabilities,  (b)  adversary  tactics,  and  (c)  manipulation  of  target 
capabilities. 

(1)  Maneuvering  capabilities.  This  refers  to  adversaries 
that  respond  as  hostile  aircraft  employing  BVR  missiles,  using 
tactics  and  missile  parameters  "similar  to"  those  likely  to  be 
encountered  in  real-world  combat  situations. 

(2)  Adversary  tactics.  Users  requested  that  adversarial 
capabilities  resemble  known  aggressor  tactics  in  unclassified 
versions.  This  was  requested  to  permit  AFRES  pilots  to  practice 
against  realistic  BVR  threats. 

(3)  Manipulation  of  target  capabilities.  As  the  third 
aspect  of  improved  target  capabilities,  users  requested  a  means  of 
configuring  the  type  and  complexity  of  targets  beyond  those 
contained  in  the  student  instructor  control  system  (SICS) .  This 
would  permit  pilots  to  select  the  type  and  level  of  tactics 
exhibited  by  the  adversary  as  well  as  the  number  of  opposing 
aircraft  engaged  in  the  sortie. 

c.  Addition  of  a  "real  aircraft"  radar  presentation.  For  the 
continuation  training  role,  the  trainer  should  present  a 
realistically  degraded  radar  image,  with  anomalies  such  as  false 
targets,  noise,  limited  range,  track  while  scan  error  rates,  and 
similar  problems  as  encountered  in  the  aircraft.  None  of  these 
kinds  of  anomalies  appears  in  the  "idealized  radar"  version 
displayed  by  the  AIT. 

d.  Addition  of  realistic  HUD  imagery.  This  enhancement  is  a 
corollary  to  the  realistic  radar  described  above.  The  head-up 
display  would  more  closely  correspond  with  what  the  pilots  sees  in 
the  aircraft  HUD.  This  would  include  such  visual  effects  as 
"jitter"  of  the  target  designator  box  during  intercepts,  low  fuel 
warnings,  noise  effects,  and  missile  numbers. 

e.  Addition  of  "real  aircraft"  flight  model.  In  the  AIT 
training  scenarios,  the  simulated  aircraft  was  equipped  with 
unlimited  missiles  and  fuel,  and  highly  "optimistic"  aircraft 
performance  capabilities.  The  performance  of  the  operational 
aircraft  is  influenced  by  changes  in  gross  weight,  drag,  center  of 
gravity,  etc.  due  to  variations  in  stores  and  fuel.  The  purpose  of 
this  enhancement  would  be  to  make  the  trainer  flight  model  more 
responsive  in  order  to  reflect  more  realistic  changes  in  aircraft 
performance  capabilities. 
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f .  Addition  of  real-time  missile  fiv-out  capability. 

This  fly-out  capability  would  be  as  close  to  real-time  as  possible 
and  would  use  classified  weapons  systems  data. 

g.  Addition  of  the  Stores  Management  Set  (SMS1  functions.  The 
SMS  permits  the  pilot  to  select  and  control  various  weapons 
delivery  parameters  via  the  multifunction  displays.  This 
capability  in  the  trainer  would  significantly  enhance  tactical 
realism  for  the  continuation  training  role. 

h.  Addition  of  Advanced  Medium  Ranee  Air-to-Air  Missile 
(AMRAAM)  capability.  This  enhancement  was  primarily  an  "nice-to- 
have"  item  rather  than  a  required  training  capability.  It  is 
related  to  the  missile  flyout  capability.  The  addition  of  the 
AMRAAM  would  likely  enhance  the  acceptance  of  the  training  device 
among  experienced  F-16  pilots. 

Summary  of  Enhancements.  Conceptually,  the  device  with  the 
above  enhancements  would  be  more  appropriately  termed  an  air  radar 
tactics  trainer,  rather  than  an  air  intercept  trainer.  The  design 
objective  of  the  above  list  of  enhancements  would  take  the  AIT  well 
beyond  the  scope  of  its  original  design.  Obviously,  since  these 
capabilities  have  not  yet  been  developed,  the  issue  of  whether  they 
would,  in  fact,  enable  AFRES  pilots  to  achieve  desired  training 
goals  can  only  be  determined  when  and  if  they  are  developed  and 
tested  in  an  operational  environment. 

To  put  the  AIT  development  issue  in  a  wider  perspective,  recall 
when  AFRES  began  experimenting  with  the  AITs,  there  were  three 
areas  of  training  need:  (a)  emergency  procedures,  (b)  instrument 
landings,  and  (c)  air-to-air  operations.  Since  the  AIT  only 
addressed  part  of  the  third  need,  the  intention  had  been  to 
eventually  develop  a  training  device  capable  of  meeting  all  three 
requirements. 

In  this  regard,  the  technology  of  the  AIT  and  the  experience 
using  it  in  the  AFRES  squadrons  became  a  point  of  departure  for 
developing  the  new  trainer.  Planning  and  development  for  the  new 
device,  called  the  MultiTask  Trainer  (MTT) ,  began  soon  after  AITs 
were  fielded  in  the  AFRES.  MTT  technology  was  a  direct  extension 
of  AIT  design  concepts  and  computer  architecture.  The  MTT  design 
evolved  rapidly  to  the  point  of  becoming  the  functional  equivalent 
to  the  F-16  Operational  Flight  Trainer  (OFT)  with  less  visual 
system  capability.  Thus,  the  design  enhancements  listed  above,  as 
identified  by  AFRES  pilots  for  the  AIT,  were  more  logically  and 
efficiently  embodied  in  the  development  of  the  MTT.  As  an 
interesting  use  of  technology,  however,  the  full  software 
capabilities  of  the  MTT  became  available  for  use  in  the  advanced 
version  of  the  AIT  which  became  known  as  the  AIT  Plus  and 
incorporated  additional  computer  capacity  to  run  the  OFT  software. 
The  present  plan  is  to  use  the  MTT  as  the  principal  training  device 
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in  the  AFRES  units  with  AIT  Pluses  as  adjunct  devices  for  sons 
aspects  of  training. 

5.  Use  of  Two-Ship  Mode.  Data  from  the  in-unit  surveys  showed  that 
only  about  one  fourth  of  AFRES  pilots  reported  using  the  networked 
two-ship  mode.  This  capability  was  designed  to  enable  the 
engagement  of  one  or  more  simulated  enemy  targets  by  one  pilot  in 
one  AIT  acting  as  flight  lead  and  another  pilot  in  the  other  AIT 
acting  as  the  wingman.  In  some  cases,  two  pilots  used  the  two-ship 
mode  to  fly  against  each  other  as  adversaries.  This  use  of  the 
device  was  regarded  mostly  as  a  novelty,  with  little  useful 
training  value,  because  operational  sorties  are  based  upon  two-ship 
operations.  Analysis  of  pilot  opinion  from  interviews  and 
questionnaires  showed  that  the  lack  of  pilot  enthusiasm  for  the 
two-ship  mode  was  mainly  from  the  lack  of  ability  to  accomplish 
much  meaningful  training  with  it  because  of  the  limited 
capabilities  of  AIT  scenarios.  Further,  to  fly  the  two-ship  mode, 
two  pilots  had  to  schedule  time  on  the  device  jointly.  Pilots 
reported  frequent  trouble  getting  the  two  trainers  to  operate 
reliably,  and  such  attempts  at  training  went  unrewarded.  Many 
pilots  reported  the  value  of  the  "two-ship"  was  not  worth  their 
time  or  inconvenience.  Better  reliability  of  the  device  was 
needed,  but  in  the  final  analysis,  this  would  not  have  solved  the 
basic  problem  of  training  value  inherent  in  the  trainer. 

6.  Trainer  Reliability  and  Technical  Service,  a  number  of  trainer 
reliability  problems  surfaced  with  AIT  deliveries  to  the  first 
AFRES  squadrons.  These  problems  included  initializing  (booting  up) 
trainers,  miscellaneous  radar  malfunctions,  overheating,  and 
several  mechanical  problems.  The  overheating  problem  turned  out  to 
be  persistent  and  required  a  design  change  in  the  AIT  hardware.  As 
training  capabilities  were  added  to  the  AIT  from  the  original 
prototype,  the  number  of  computer  boards  and  other  internal 
components  increased,  adding  to  the  heat  load  of  the  device  and  of 
the  room  in  which  it  was  operated.  To  provide  improved 
reliability,  it  was  necessary  to  redesign  the  cabinet  interior  to 
provide  additional  cooling  capacity. 

From  the  AL/HRA  perspective,  there  were  several  requirements  in 
minimizing  equipment  reliability  problems  for  users.  The  first, 
and  the  most  important,  was  to  adopt  a  user-oriented  view  of  device 
reliability.  The  second  was  to  initiate  and  maintain  frequent 
contact  with  the  user.  The  third  was  to  identify  and  correct 
problems  as  expeditiously  as  possible:  the  fourth,  to  document 
problems  completely;  and  the  fifth,  to  follow-up  with  users  to 
verify  resolution  of  problems. 

7.  Technology  Transition.  AL/HRA  began  transitioning  AIT 
technology  to  the  Training  System  Center  of  the  Air  Combat  Command 
at  Luke  AFB  in  1989.  The  basic  agreement  was  that  TSC  would  build 
the  devices  using  hardware  specifications  and  software  provided  by 
AL/HRA.  The  original  intention  had  been  for  complete  transition  of 
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AIT  technology,  that  is,  all  hardware  and  software  functions  so 
that  TSC  would  be  able  to  assume  responsibilities  for  device 
manufacture,  installation  at  user  sites,  maintenance  and  repair. 
Under  this  plan,  AL/HRA  would  continue  to  develop  and  test  new 
software  and  transition  it  to  TSC. 

Up  to  the  advent  of  the  AIT,  TSC  had  been  involved  primarily  in 
the  development  of  training  hardware,  having  produced  a  number  of 
part-task  practice  devices  which  duplicated  certain  avionics 
functions  of  aircraft.  Examples  of  F-16  training  devices  included 
a  Fire  Control  and  Navigation  Panel  (FCNP)  Trainer  and  a  Store 
Management  Set  (SMS)  trainer. 

The  transition  of  the  hardware  function  was  successfully 
assumed  by  TSC.  Replication  of  equipment  cabinets,  assembly  of 
cockpit  components,  instruments,  controls,  and  integration  of 
computer  hardware  were  well  accomplished.  TSC  was  able  to  make  a 
number  of  improvements  in  the  design  and  function  of  trainer 
hardware,  including  stick  and  throttle  and  cabinet  cooling 
functions . 

However,  the  AIT  presented  a  substantial  challenge  to  TSC  in 
terms  of  software  transition  and  maintenance.  The  complexity  of 
AIT  software  was  considerably  beyond  any  of  the  systems  previously 
manufactured  by  TSC.  The  level  of  software  experience  was  less 
than  adequate  for  this  aspect  of  the  transition.  Difficulties 
became  quite  apparent  when  AL/HRA  began  transferring  software 
upgrades.  The  laboratory  standard  for  software  "readiness"  was 
apparently  less  stringent  than  TSC  personnel  were  willing  to 
accept.  So  a  period  of  accommodation  was  required  to  work  out 
differences  and  agree  on  standards  for  software  transition.  Morr 
difficult  was  the  issue  of  software  maintenance.  It  became  clear 
that  TSC  was  not  in  a  position  to  maintain  software.  The  result 
was  that  rather  than  affecting  a  complete  transition  of  AIT 
technology  to  ACC/TSC,  AL/HRA  retained  responsibility  for  device 
software  maintenance.  TSC  as  the  point  of  contact  for  AIT 
maintenance,  took  the  lead  when  field  users  (primarily  AFRES  units) 
required  services.  Under  this  arrangement  TSC  diagnosed  needed 
repairs  and  when  problems  with  software  were  identified,  TSC 
referred  them  back  to  AL/HRA.  During  early  fielding  of  devices  to 
AFRES  units,  a  large  number  of  device  malfunctions  were 
attributable  to  software  problems.  The  division  of  hardware  and 
software  maintenance  responsibilities  proved  to  be  cumbersome  and 
resulted  in  service  delays  because  of  the  number  of  coordinations 
required  to  repair  AITs  (see  related  material  in  6.  Trainer 
Reliability  and  Technical  Service,  above) . 

8.  ANG  F-16  Conversion  Training.  In  1990,  approximately  14  Air 
National  Guard  squadrons  were  identified  for  conversion  from 
existing  operational  aircraft  to  the  F-16.  The  conversion 
schedules  spanned  a  period  of  about  three  years.  Squadrons  were 
sequenced  through  the  F-16  conversion  course  which  operated  at  the 
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184th  FS,  McConnell  AFB,  KN.  The  ANG  schoolhouse  made  extensive 
use  of  the  AIT  in  its  conversion  course  syllabus.  Three  AITs  were 
located  at  the  squadron  to  support  training.  Evaluation  of  AIT  use 
at  the  ANG  schoolhouse  confirmed  that  benefits  from  the  trainer 
were  essentially  the  same  as  those  observed  in  the  ACC  schoolhouses 
at  Luke  and  MacDill  AFBs. 

Several  converting  units  also  made  use  of  the  AIT  to 
familiarize  pilots  with  air  intercepts  prior  to  attending  the 
conversion  course  at  McConnell  AFB.  The  first  squadron  to  use  the 
device  for  this  purpose  was  the  120th  FS  at  Buckley  ANG  Base,  CO. 
To  implement  the  trainer,  a  special  in-unit  training  syllabus  was 
developed  around  the  capabilities  of  the  AIT.  All  unit  pilots 
received  up  to  six  hours  of  training  on  the  device  prior  to 
attending  the  conversion  course.  Pilot  grade  slip  records  examined 
at  the  schoolhouse  showed  that  pilots  who  received  the  AIT  training 
prior  to  the  course  received  significantly  higher  instructor 
ratings  on  simulator  and  aircraft  air-to-air  training  sorties 
compared  to  pilots  who  had  not  received  the  AIT  training. 

Following  the  demonstrated  success  of  the  squadron-based  AIT 
training,  several  other  ANG  units  obtained  AITs  for  use  by  their 
pilots  prior  to  attending  the  conversion  course.  some  of  these 
units  were  able  to  obtain  AITs  on  loan  from  AFRES  units. 


REVIEW  OF  LESSONS  LEARNED  BY  TECHNOLOGY  TRANSITION  ISSUES 

At  the  beginning  of  this  report,  a  list  was  provided  of  several 
issues  to  be  addressed  as  part  of  the  Study  of  AIT  technology 
transition.  The  purpose  of  this  final  part  of  the  report  is  to 
group  the  lessons  learned  according  to  their  relevance  to 
transition  issues.  The  headings  listed  below  are  the  transition 
issues  with  pertinent  lessons  learned  summarized  in  single 
paragraphs  below  each  heading.  In  some  cases,  lessons  learned  may 
be  listed  under  more  than  one  transition  issue  depending  upon 
relevance  to  more  than  one  area. 

1.  Device  Acceptance  bv  Users 

The  training  capabilities  and  utility  of  the  AIT  were  well 
accepted  by  the  Air  National  Guard  training  squadron  and  the  device 
became  a  key  training  resource  during  conversion  to  the  F-16A 
aircraft  from  the  A-7D  aircraft. 

The  prototype  AIT  was  not  equipped  with  a  forward  view,  out- 
the-cockpit,  visual  display.  While  the  initial  reaction  of  ANG 
student  pilots  to  this  limitation  was  negative,  the  training  effect 
was  positive.  The  "forced"  reliance  on  HUD/radar  information 
during  training  produced  substantial  skill  gains  in  intercept 
geometry  and  situational  awareness,  as  reported  by  instructor 
pilots. 
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ANG  pilots  found  the  lack  of  pilot-controllable,  simulated 
flight  of  the  prototype  AIT  unacceptable.  As  a  result,  AL/HRA 
promptly  developed  and  retrofitted  enhanced  flight  control 
capabilities. 

The  AIT  was  well  accepted  in  the  ACC  schoolhouse  community  as 
an  introductory  level  air-to-air  training  device.  It  filled  an 
important  training  need  in  the  B  course.  The  effectiveness  of  the 
AIT  and  its  acceptance  were  the  result  of  extensive  training 
analysis  and  design,  technology  development,  and  long-term  efforts 
to  improve  device  training  qualities,  including  application  of 
lessons  learned  from  experience  with  the  ANG. 

Instructor  pilots  (ACC  schoolhouse)  accepted  the  AIT  as  a 
useful  training  resource.  This  was  attributable  to  the  AIT's 
training  software  capabilities,  compatibility  with  syllabus 
objectives  and  content,  and  the  convenience  to  students  afforded  by 
the  presence  of  the  AIT  in  the  academic  squadron. 

2.  Usage  Methods  and  Patterns 

ANG  pilots  required  substantially  improved  training  in 
HUD/radar  interpretation  skills  as  a  lead-in  to  the  AIT  phase. 
Lack  of  these  enabling  skills  caused  AIT  training  to  be  much  less 
efficient  than  it  should  have  been  during  the  first  several 
classes.  This  led  to  revision  of  the  training  syllabus  to  provide 
more  AIT  time  for  basic  skills  development  and  eventually  led  to 
the  development  of  additional  training  materials. 

Assessment  of  pilot  performance  of  intercepts  was  better 
accomplished  using  well-developed  debriefing  feedback  techniques, 
rather  than  a  computer  scoring  approach.  The  development  of  a 
linear  scoring  system  which  assigned  weighted  values  for  various 
intercept  components  proved  inadequate  for  purposes  of  training 
evaluation. 

The  training  utility  of  the  AIT  for  switchology  skills 
maintenance  corresponded  with  pilot  experience  (in  AFRES 
squadrons) . 

ANG  squadrons  converting  to  the  F-16  found  the  AIT  advantageous 
as  a  familiarization  device  in  squadrons  and  as  part  of  the 
conversion  course  syllabus  at  the  schoolhouse.  Some  of  the  AFRES 
units  relinquished  their  AITs  for  use  by  ANG  units. 

3.  Design  Features  and  Options 

ANG  pilots  found  the  lack  of  pilot-controllable,  simulated 
flight  of  the  trainer  unacceptable.  As  a  result,  AL/HRA  promptly 
developed  and  retrofitted  enhanced  software  to  provide  acceptable 
flight  controls.  Device  training  utility  and  pilot  acceptance 
increased  following  this  upgrade.  The  resulting  lesson  for  AL/HRA 
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was  that  any  aircraft  capabilities  included  in  the  simulation  must 
be  functionally  equivalent  to  the  aircraft.  Or,  if  present  in  the 
device,  they  must  be  of  such  obviously  low  fidelity  that  pilots  are 
not  led  to  equate  trainer  capabilities  with  those  of  the  aircraft. 

Student  pilots  (in  the  ACC  schoolhouse)  were  able  to 
successfully  use  the  training  menus,  instructional  features,  and 
practice  scenarios  without  the  assistance  of  an  instructor  pilot. 
Students  generally  preferred  the  self-paced  mode  of  training  on  the 
AIT  as  a  means  toward  developing  intercept  skills.  Instructors 
also  accepted  the  self-paced  mode  of  use. 


To  support  AFRES  operations  squadrons,  device  capabilities 
would  extend  well  beyond  those  of  the  AIT.  In  view  of  this 
finding,  the  magnitude  of  needed  changes  and  associated  costs,  the 
reasonable  choice  would  be  a  new  trainer  design,  rather  than  a 
modified  AIT. 

Although  the  networking  capability  of  two  AITs  for  two-ship 
operations  was  developed  as  a  training  enhancement  for  the  AFRES 
squadrons,  pilots  in  these  units  made  little  use  of  this  feature. 
Related  to  this  finding  was  the  fact  that  pilots  preferred  to  use 
the  AIT  by  themselves.  Pilots  also  reported  having  problems  using 
the  two-ship  mode. 

4.  Training  Capabilities  and  Effectiveness 

ANG  students  required  considerable  instructor  assistance  to 
learn  basic  operation  of  the  AIT.  However,  once  students  were 
familiar  with  the  equipment  and  could  "navigate"  within  the 
software,  self-paced  practice  on  the  device  became  a  realistic 
option. 

The  prototype  AIT  was  not  equipped  with  a  forward  view  out-of- 
the-cockpit  visual  display.  While  the  initial  reaction  of  ANG 
student  pilots  to  this  limitation  was  negative,  training 
effectiveness  was  positive.  The  resulting  "forced"  reliance  on 
HUD/radar  information  during  training  produced  substantial  skill 
gains  in  intercept  geometry  and  situational  awareness,  as  reported 
by  instructor  pilots. 

For  experienced  F-16  operational  pilots  in  the  AFRES 
operational  squadrons,  the  training  utility  of  the  AIT  was  limited. 
Even  with  enhancements  in  the  capabilities  of  the  trainer,  pilots 
were  able  to  realize  relatively  minor  training  benefits  from  the 
AIT  such  as  basic  switchology  skill  refreshment. 

Limitations  in  AIT  fidelity  and  training  capabilities  which 
were  acceptable  in  the  schoolhouse  were  found  to  be  unacceptable  to 
operational  pilots  in  AFRES  units,  particularly  in  units  where 
pilot  experience  was  high.  Data  collected  in  these  squadrons 


32 


showed  that  once  a  pilot  had  accrued  about  300  hours  in  the 
aircraft,  the  value  of  AIT  training  became  negligible. 

5.  Device  Reliability 

The  operational  reliability  of  the  AIT  from  its  beginnings  ANG 
had  been  exemplary.  However,  as  the  trainer  evolved  and  its 
complexity  increased,  so  did  reliability  problems.  By  the  time  the 
AITs  had  evolved  to  the  AFRES  application  stage,  reliability 
problems  had  become  considerable  and  required  special  attention. 
Better  technical  service  to  users,  particularly  during  the  initial 
period  of  use  at  the  unit,  could  have  alleviated  trainer 
reliability  problems  and  promoted  better  user  acceptance  in  the 
squadrons.  This  was  especially  true  for  the  first  AFRES  squadrons 
who  received  the  AITs. 

6.  Technology  Development  Process 

The  development  and  delivery  of  timely,  yet  state-of-the-art 
training  technology  directly  from  AL/HRA  to  the  user  presented  a 
dilemma.  The  prototyping  process  had  to  accommodate  timely 
infusion  of  new  technology,  otherwise,  the  trainer  would  be 
obsolete  upon  delivery.  However,  if  technology  infusion  dominated 
development,  the  effort  would  become  consumed  "chasing"  technology 
so  that  a  usable  device  would  never  be  delivered.  A  balance  was 
reached  by  developing  a  two-step  prototyping  process  which  made 
possible  timely  infusion  of  technology  into  the  trainer  design. 

7.  Technology  Transition  Process 

Findings  from  an  initial  AIT  empirical  evaluation  ANG  training 
were  inconclusive  relative  to  AIT  training  effectiveness. 
Subsequent  trainer  evaluation  required  substantially  improved 
experimental  methods  and  procedures  to  insure  useful  and  reliable 
results . 

Fully  transitioning  the  prototype  AIT  from  the  laboratory  to 
the  user/manufacturer  was  not  possible  because  of  inability  to 
assume  responsibilities  for  software  development,  upgrades,  and 
maintenance. 

ANG  F-16  squadrons  found  the  AIT  advantageous  as  a 
familiarization  device  in  the  squadrons  and  as  part  of  the 
conversion  course  syllabus  at  the  schoolhouse.  Some  of  the  AFRES 
units  relinquished  their  AITs  for  use  by  ANG  units. 
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